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We study how the custodial symmetry in the Higgs sector of the Georgi-Machacek (GM) 
model can be tested at the LHC. As the minimal extension of the Higgs triplet model, 
in which tiny neutrino masses are generated via the Type-II Seesaw Mechanism, the GM 
model keeps the electroweak p parameter at unity at tree level. In the GM model, there 
arc 5-plct (J?s), 3-plct (H3) and singlet (Hi) Higgs bosons under the classification of the 
custodial SU(2)v symmetry, in addition to the standard model-like Higgs boson (h). These 
new Higgs bosons have the following characteristic features at the tree level: (1) the masses 
of the Higgs bosons belonging to the same SU(2)v multiplet are degenerate; and (2) H5 
and Hi couple to the electroweak gauge bosons but not SM quarks, whereas H3 couples to 
the quarks but not the gauge bosons. Wc find that the H§ production from the weak vector 
boson fusion process and the Drell-Yan process associated with H3 are useful in testing the 
custodial symmetry of the Higgs sector at the LHC. In addition, these processes can also be 
used to discriminate from other models that contain singly-charged Higgs bosons and extra 
neutral Higgs bosons. We also investigate a possible enhancement in the h — > 77 as well as 
h — > Z7 decays. 



I. INTRODUCTION 

Recently, a new particle of mass about 125 GeV has been discovered at the CERN Large Hadron 
Collider (LHC) with a total production rate consistent with that of the standard model (SM) Higgs 
boson [l|,Q]. Confirming this Higgs-like particle as the one responsible for the electroweak symmetry 
breaking is of paramount importance in particle physics because, for one thing, it explains the origin 
of mass for elementary particles. While further detailed examinations are required, the current LHC 
data show some deviations in the pattern of its decay branching ratios from the SM expectation. 
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This leads to the speculation that the Higgs sector may not as simple as the one in SM. 

In certain new physics models such as supersymmetry, the Higgs sector has to be extended with 
additional nontrivial isospin SU(2)l scalar multiplets for consistency or to explain new phenomena. 
Such an extension also holds the capacity to provide additional CP-violating sources for low-energy 
phenomena as well as baryon asymmetry of the Universe. For example, the two-Higgs doublet 
model (2HDM) [3| is an extensively studied prototype in which an additional scalar doublet is 
introduced. SU (2) l triplet Higgs fields also occur in some new physics models, such as the left- 
right symmetric model [4] and little Higgs models Q|. By introducing a complex triplet Higgs 
field, it is possible to have an effective dimension-5 operator for generating tiny Majorana mass 
for neutrinos. Therefore, it is important to determine the true Higgs sector in order to exactly 
know what kind of new physics models exist at the TeV or higher energy scales. In this paper, 
we want to focus on the phenomenology of the extended Higgs sector in the model proposed by 
Georgi and Machacek (GM) (| in mid-80s. We investigate how one can distinguish it from the 
other Higgs-extended models at the LHC. 

The GM model contains a Higgs doublet field $ and a triplet field A, with the latter containing 
a hypercharge Y = 1 component and a Y = component. The model is of great interest because 
it can provide tiny mass to neutrinos a la the Seesaw Mechanism, dubbed the Type-II Seesaw Q]. 
Moreover, it has been shown that the Higgs potential in this model can be constructed to maintain 
a custodial SU(2)y symmetry at the tree level sj], keeping the electroweak p parameter at unity 
to be consistent with the experimental constraint. In the model, there are 5-plet Higgs bosons 
H5 (=H^ 1± ,H^ : ,H^), 3-plet Higgs bosons H% (=H^,H®) and singlet Higgs boson H® under the 
classification of the SU{2)y symmetry. The masses of the Higgs bosons belonging to the same 
SU(2)v multiplet are the same at the tree level as the consequence of custodial symmetry. 

The doubly-charged Higgs boson for example, is an important but not unique feature of 

the model. Finding particles in one Higgs multiplet and checking their (near) mass degeneracy 
would better verify the model. Strategies of discovering such Higgs bosons, however, depend largely 
on the vacuum expectation value (VEV) of the Higgs triplet field, i>a- 

In the minimal Higgs triplet model (HTM) where only one additional complex Higgs triplet is 
introduced, the doubly-charged Higgs bosons couples dominantly to a pair of like-sign leptons when 
VA < 10 -4 GeV. The collider phenomenology of this scenario has been extensively studied recently 
9|j. The doubly-charged Higgs boson has been searched for at the Tevatron [l(| and the LHC 



111 ] by looking for like-sign lepton pairs with the same or different flavors. A lower mass bound 



of about 400 GeV has been obtained for most scenarios. On the other hand, the doubly-charged 
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Higgs bosons couples dominantly to a pair of like-sign W bosons when v& > 10 -4 GeV 1 . This 
possibility is less explored experimentally. Besides, the triplet VEV in the HTM is constrained by 
the p parameter to be less than a few GeV, limiting significantly the discovery reach at the LHC. 

In the GM model, a larger triplet VEV is allowed due to the custodial symmetry. It is there- 
fore interesting to consider signatures of the like-sign gauge boson decays. In Ref. 12||, collider 



phenomenology of the GM model has been discussed in the case of light triplet-like Higgs bosons, 
e.g., less than 100 GeV. A recent study by one of the authors and collaborators [l3] finds that with 

— 55 GeV and appropriate cuts, the current LHC can reach up to 450 GeV for the doubly- 
charged Higgs mass. In this work, we further explore consequences of the custodial symmetry in 
the Higgs sector of the GM model and study the phenomenology of its entire Higgs sector at the 
LHC. We find that the single production of H§ via the weak vector boson fusion process is useful 
to test the mass degeneracy among the H§ bosons. We also find that the Drell-Yan process, where 
H§ and H3 are simultaneously produced can be used to check the mass degeneracy among #3. 

The structure of this paper is organized as follows. We review the GM model in Section [TIJ The 
Higgs bosons are first classified according to their group representations under the custodial sym- 
metry. We then consider possible mixings between the two triplets and between the two singlets, 
and work out their masses. A mass relation among the Higgs bosons of different representations 
is obtained in the decoupling limit when the triplet VEV vanishes. Finally, we show the Yukawa 
couplings between SM fermions and the physical Higgs bosons. In Section IIIH we consider both 
theoretical constraints of perturbative unitarity and vacuum stability and the experimental con- 
straint from the Z-pole data of Z — > bb decay at one- loop level. In particular, they impose bounds 
on the triplet VEV and the Higgs triplet mass. In Section llYl we discuss in detail how the Higgs 
bosons decay in scenarios with or without hierarchy in the masses of the physical Higgs singlet, 
3-plet, and 5-plet. The collider phenomenology of the model can be drastically different in different 
regions of the v^-Am (Am is the mass difference between H§ and H3) space. Section IVl discusses 
how the Higgs bosons can be searched for at the LHC. Finally, we compute the decay rates of 
h — > 77 and Z7 in the model in Section IVI1 Our findings are summarized in Section IVII1 



1 When there is a non-zero mass splitting among the scalar bosons in the triplet Higgs field and the doubly- 
charged Higgs boson mass is the heaviest, the cascade decays of the doubly-charged Higgs boson become dominant. 
Phenomenology of this scenario has been discussed in Refs. [lj. 
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II. THE MODEL 

In the GM model, the Higgs sector is composed of the SM isospin doublet Higgs field (f> with 
hypercharge Y = 1/2 and two isospin triplet Higgs fields x with Y = 1 and £ with Y = 0. These 
fields can be expressed in the form: 

x- e x + , (i) 

where $ and A are transformed under SU(2) L x SU{2) R as <I> ->■ U L $U R and A ->■ U L AU R with 
^7l,.r = ex P{i@L rT 11 ) and T a being the SU(2) generators. The neutral components in Eq. ([T]) can 
be parametrized as 

<f>° = -^(fo + Vj + ifc), x° = ~^(Xr + iXi) + v x , £° = £r+^, (2) 

where v^, v x and are the VEV's for cfp , x° an d £°, respectively. When the two triplet VEV's v x 
and are taken to be the same, i.e., v x = = ua, the SU{2)l x SU(2)r symmetry is reduced to 
the custodial SU(2)y symmetry. The phase convention for the component scalar fields are chosen 

to be x~- = (x ++ r, 4>- = -(<t> + y, x- = -(x+r, r = -(t + r and e = (ey. 

The relevant Lagrangian involving the Higgs fields can be written as 

-Cgm = £kin + £y + C-v — Vh, (3) 

where Adm >Cy, C u and Vh are the kinetic term, the Yukawa interaction between (f> and the 
fermions, the neutrino Yukawa interaction between x and the lepton doublets, and the Higgs 
potential, respectively. 

The most general Higgs potential invariant under the SU(2)i x SU{2)r x U(1)y symmetry in 
terms of the fields defined in Eq. ([1]) is 

V H = m?tr($t$) + m 2tr(AtA) + A 1 tr($ t $) 2 + X 2 [ti(A^A)} 2 + A 3 tr[(AtA) 2 ] 
+ A 4 tr($ t $)tr(A t A) + A 5 tr ( $t y^y) ti(Ah a At b ) 

+ /iitr ( ptAP ) a& + ^ 2tr ( At ^ A * b ) (4) 

where r a are the Pauli matrices, t a are the 3x3 matrix representation of the SU (2) generators 
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and the matrix P is defined as 



( -1/V2 i/y/2 ^ 
P= 1 ■ (6) 
\ l/v/2 i/>/2 / 

As in the HTM, the SM electroweak symmetry breaking can induce the triplet field to develop 
a VEV va through the [i\ term in the Higgs potential. To our knowledge, most of the previous 
analyses ignore both \i\ and [12 interactions in their phenomenology studies. We will keep these 
terms in this work. 

Using the tadpole conditions, 



0. 



dV H 



0, 



dV H 



dXr 



0. 



(7) 



the parameters m\ and m 2 , can be eliminated as 



m-. 



rrin 



3 3 



1 



1 



-s 2 M 2 
-s H M l , 



1 



-y H \ 2 + -rsjjXa + c&A 4 + -c z H X 5 + -cjjMi + -M|, 



1 



(8a) 
(8b) 



where v 2 = v \ + 81^ = \/{\/2Gf) and tan## = 2y/2v&/v ( j > with s# = sin#fl- and ch = cos9h- In 
Eq. dHJ), we introduce M 2 and Mf as 



Mi, 



Ml 



(9) 



The second and third conditions in Eq. ((7J) give the same constraint in Eq. (|8bj) as long as f x = 

Before we discuss the mass matrices and the mass eigenstates for the Higgs bosons, it is conve- 
nient to classify the Higgs boson states according to the custodial SU(2)y symmetry. The triplet 
field A, which can be understood as a 3<S> 3 representation of the SU(2)y multiplet, can be decom- 
posed into the irreducible representations 5 © 3 © 1. Likewise, the doublet field <3? being the 2 © 2 
representation of the SU(2)y multiplet, can be decomposed into 3 © 1. The 3 representation of 
<]? can be identified as the Nambu-Goldstone (NG) bosons of the SM as long as there is no mixing 



between the 3 representations of A and <E>. The 5-plet (iJ 5 , and H§), the 3-plet (H 3 and 
H$) and the singlet (H%) originating from A can be related to the original component fields as 



1 



H° 3 = XU 



(10) 



It is seen that H® and H\ are CP-even states, whereas H$ is a CP-odd state. In Eq. (llOh . the 
scalar fields with a tilde are not mass eigenstates in general. They can in principle mix with the 
corresponding scalar fields from the Higgs doublet field. 



The mass of the doubly-charged Higgs boson H 5 ^ is 



sj I \B--cj I X 5 )v' + ci I Mt + Ml 



(11) 



The mass matrix for the CP-odd Higgs states in the basis of H®) and that for the singly-charged 
states in the basis of (<ft + , H% , H§) are given by 



(M 2 ) CP -odd 



-X 5 v 2 - M 2 




'11 



-CRSH 



-CRSH 



H 



(M 2 ) ± 



(M 2 ) CP-odd 








\ 



V 







m 2 Hr j 



The mass matrix for the CP-even Higgs states in the basis of (4> r , Hy, H®) is 

( (M 2 ) n (M 2 ) ]2 \ 

) CP-even — 



(M 2 ) 12 (M 2 ) 22 



112 
)22 



V 











where the elements of the 2x2 submatrix are 



(M 2 ) n = Sc^Ai^ 2 , 

(M 2 ) 22 = 4(3A 2 + X 3 )v 2 + c 2 H M\ 



-M. 



2 ! 



(M 2 



112 



^s H c H [{2X i + X 5 )v 2 - M 2 ] 



(12a) 
(12b) 

(12c) 



The mass eigenstates are related to the above-mentioned states via the following unitary trans- 
formations 



^3° 



^CP- 



odd 



G° 



H° 3 



Hi 



u± 



Hi 

\ H ij 



Hi 



U C p- 



' h ' 
H\ 



(13) 
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where and G° are the NG bosons for the longitudinal components of the and Z bosons. 
The explicit forms of the unitary matrices are 



/ 



^CP-odd 




u± 







\ 



^CP- 



odd 







V 



C/cp-, 



1 

where c a = cos a, s a = sin a and the mixing angle a is defined by 

)12 



c a s a 
s a c a 
1 



(14) 



tan 2a 



2(M 2 ^ 



(M 2 ) n - (M2) 22 • (15) 
The masses of the singly-charged Higgs bosons (H§ and H 3 ), the CP-odd Higgs boson (H®) and 
the CP-even Higgs bosons (H®, H® and h) are then 

2 2 2 2 2 1 \ 2 1 71^-2 

m +=m H o=m ++, m + = m H o = --X 5 v +M 1 , 



771 1 



(M 2 ) n c 2 + (M 2 ) 22S 2 a + 2(M 2 ) 12SqCq , 



m^o = (M 2 ) llS 2 + (M 2 ) 22 c 2 - 2(M 2 ) 12 s a c a . (16) 

It is observed that H 5 ^, and #5 are degenerate in mass and so are and H® because of the 
custodial invariance in the Higgs potential. Therefore, the Higgs boson masses can be conveniently 
written as 



m H 5 = m H ++ = m H+ = m H°> m H 3 = m H + = m H°> m H 1 = m H°- 



(17) 



The five dimensionless couplings in the potential, Ai, . . . , A5, can be substituted by the five physical 
parameters m# 5 , m,H 3 , m h an d a as follows: 

1 



Ai 
A 2 
A3 

A 4 



8v 2 c 2 H 



1 



Qv 2 i 



1 2 2 1 1 i\ 
{m h c a + m Hl s a ), 

[2m 2 Hi c 2 a + 2m\sl + 3M 2 - 7m\ + <Sc\{jn\ z - M 2 [ 



H 



2 e 2 



[c 2 H (2M 2 -3m 2 H3 )+m 2 H5 -M 



2] > 



Qv 2 shch 



^-s 2a (m 2 h - m 2 Hi ) + 3s H CH(2m 2 H3 - Mf 



A 5 = — (M 2 -m 



H 3 , 



(18) 



The decoupling limit of this model can be obtained when we take the v a — > limit (or equiva- 
lently s/j — >■ 0). In this limit, the mass formulae of the Higgs bosons reduce to 



1 



77? 



-X 5 v 2 + M{ + M 2 , m z H3 = --X 5 v 2 + M(, m 2 Hi = M( - -M 2 , m 2 h = 8Xiv z . (19) 
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Notice that M| is proportional to sh^2, and thus it becomes zero in this limit for a fixed value of 
H2- If one wants to fix M| at a finite value, fi2 has to be taken to infinity to compensate s# — > 
and eventually violates perturbativity in this model. Therefore, Af| = is the natural choice in 
this limit. On the other hand, M\ is proportional to /xi/sjj. Even in the sh — > limit, we can 
take a finite value for M\ as long as \i\ — > at the same rate as sh- Consequently, the triplet-like 
Higgs bosons decouple when M 2 3> v 2 , and only h remains at the electroweak scale and acts like 
the SM Higgs boson. In addition, in the decoupling region v& — 0, we find a simple mass relation 
for the triplet-like Higgs bosons: 

= \ m2 H 3 ~ 2 m H 5 - ( 20 ) 

For the convenience in discussing interactions between leptons and the Higgs triplet field, we 
reorganize the Higgs fields as follows: 

Hi ^MM)- 

The relationship between the two representations in Eqs. ([T]) and (|2ip are given in Appendix A. 
With the introduction of the x field above, the Yukawa interactions between the lepton doublets 
and the Higgs triplet are 

C v = hijlJfi^xLi + h.c. (22) 

If we assign two units of lepton number to \i then the A5 and \i\ terms in the Higgs potential 
violate the lepton number. If we then take A5 = [X\ = 0, #3 becomes massless and corresponds 
to the NG boson for the spontaneous breakdown of the global U(l) lepton number symmetry. In 
fact, Hn are also massless in that case because of the custodial symmetry. 
The Major ana mass of neutrinos is derived as 

This mass matrix can be diagonalized by the Pontecorvo-Maki-Nakagawa-Sakata matrix VpmnSj 
and the Yukawa matrix hij can be rewritten as 

h = 2 ^p T MNS^ laS ^PMNS^ (24) 
VS H 

The left-handed neutrino fields are then transformed as 

"L = ^PMNS^I- ( 25 ) 
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For simplicity, we hereafter assume that Vpmns is the unit matrix and the mass eigenvalues of m^ iag 
are degenerate: m^ iag = diag(mj,, m„, m v ). In terms of the scalar mass eigenstates, the interaction 
terms are 

2V2m u 



+ 



2V2m u 

s H v 
2m,, 



s H v 



[H+ + c H H+ + s H G + ) ufP Lei 



S H V 



1 

71 



(iZg + V2s a h + c a Hl) + i(G°s H + H$c H ) 



vfPL^i + h.c. 



(26) 



The Yukawa interaction between the fermions of one generation and the Higgs doublet <fi is 
given by 



C Y = —Y u Ql4>ur - Y d Q L (j)d R - Y e L L 4)e R + h.c, 



(27) 



with <j) = iT2<f>* ■ In terms of the fermion masses rrif = ^^Yf an d the physical Higgs states, the 
interaction terms are expressed as 



an 

f=u,d,e 



—ffh - —JfHl + iSign(/) tan^/75/^3 

CH C H 



[tand H u(m u P L - m d P R )dtiJ\ H 



tanOn^PReHt + h.c, 



(28) 



where V ud is one element of the Cabibbo-Kobayashi-Maskawa (CKM) matrix, Sign(/ = u) = +1 
and Sign(/ = d,e) = —1. 

Finally, we discuss the kinetic terms for the Higgs fields 



£kin = ^tr(D M $)t(D^$) + itr(D^A)t(D^A), 



(29) 



where the covariant derivatives are 



D.A = a, A + igt a W?A - ig'B.At 3 . 



(30) 
(31) 



The masses of the gauge bosons are obtained under the condition of v x = v^ = v/\ as 

n 2 n 2 
2 y 2 2 y 2 

4 4 cos^ 0w 



(32) 



Thus, the electroweak rho parameter p = m 2 SL /(m 2 7 cos 2 6 w) is unity at the tree level. One-loop 



corrections to p have been calculated in Ref. 



151 ] for the GM model. The deviation of p from unity 



depends on the logarithm of the triplet-like Higgs boson masses and, therefore, the one-loop effect 
is not important in this model. 
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The Gauge-Gauge-Scalar (Gauge-Scalar-Scalar) vertices are listed in Table IIHI (Table IIVP in 
Appendix B. We note that there is the H^W^Z vertex at the tree level in the GM model (see 
Table IIIII) . In the Higgs-extended models with p = 1 at the tree level and having singly-charged 
Higgs bosons {e.g., the 2HDM), the H^W^Z vertex is absent at the tree level fl(| and can only be 
induced at loop levels. Therefore, the magnitude of this vertex in such models is much smaller than 
that in the GM model. Thus, this vertex can be used to discriminate models with singly-charged 
Higgs bosons. The possibility of measuring the H^W^Z vertex has been discussed in Refs. 

]j| for future linear colliders. 



for the LHC and in Ref. 



17] 



III. CONSTRAINTS 

In this section, we discuss constraints on the parameter space of the GM model. First, we 
consider the theoretical constraints from perturbative unitarity and vacuum stability. Secondly, as 
experimental constraints, we consider the Zbb data and other B physics data. 

A. Perturbative unitarity and vacuum stability bounds 



19l | and can be 



The perturbative unitarity bound for the GM model has been studied in Ref. 
directly applied to our analysis. Before doing so, we will make a change in the parameterization. 
This is because Eqs. (|18p suggest apparent divergences in A2,3,4 in the limit va <C v. However, this 
is only an artefact that can be avoided by reparameterization. 

We therefore select the following parameterization 

rn 2 Hl = \{3m 2 H3 -m 2 H5 +3s 2 H M 2 ), M 2 = I (3m 2 H3 - m 2 Hs + M 2 ) , M 2 = M 2 , (33) 

in terms of which all the dimensionless couplings can be rewritten for sin a = as 

ml ml, - m 2 „ + M 2 + M 2 m 2 „ - M 2 

A l — o o 9 i A 2 — ^5 ) A 3 



m 2 H3 +m 2 H5 -M 2 m^-m^+M 2 

A 4 = ^2 > A 5 = ~2 • ( 34 ) 

It is seen that the va dependence drops out in A2,3,4 and no divergent A's appear even when va *C v. 

For the vacuum stability condition, we require that the potential is bounded from below in any 
direction with large scalar fields. This condition imposes constraints on the dimensionless coupling 
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FIG. 1: Constraints from the unitarity and vacuum stability in the M-m,H b plane. In all the plot, the 
uncolored regions are allowed, and the 3-plet Higgs mass is taken to be 150 GeV, i>a = 1 MeV and a = 0. 
Blue, gray and pink shaded regions are respectively excluded by the vacuum stability bound, unitarity bound 
and a negative singlet Higgs mass {mn 1 < 0). The left, center and right plot show the case of M = 0, 300 
GeV and 350 GeV, respectively. 



M = 0, m H1 = 300 GeV, v 4 = 1 MeV, a = M = 200 GeV, = 300 GeV, v 4 = 1 MeV, a = M = 230 GeV, m H , = 300 GeV, v 4 = 1 MeV, a = 




m H , [GeV] n. [GeV] m m [GeV] 



FIG. 2: Constraints from the unitarity and vacuum stability in the M-mu b plane. In all the plot, the 
uncolored regions are allowed, and the 3-plet Higgs mass is taken to be 300 GeV, va = 1 MeV and a = 0. 
Blue and pink shaded regions are respectively excluded by the vacuum stability bound and the unitarity 
bound. The left, center and right plot show the case of M = 0, 200 GeV and 230 GeV, respectively. 

constants Ai, . . . , A5. In the GM model, we then derive the following inequalities 

Ai>0, A2 + A3X), A 2 + ^A 3 >0, -|A 4 | +2 v / Ai(A 2 + A 3 ) > 0, 

A4-^|A5| + v / 2Ai(2A2 + A 3 ) > 0. (35) 

They have taken into account the positivity of all combinations of two non-zero scalar fields, as 
have been discussed in Ref. [20I ] for the HTM. 

Fig. Q] shows the regions excluded by the unitarity and the vacuum stability constraints for the 
case of mH 3 = 150 GeV and va = 1 MeV. The left, center and right plots show the cases for M=0, 
300, 350 GeV, respectively. For the unitarity bound, we consider the 5-wave amplitudes for elastic 
scatterings of two scalar boson states and require their absolute values of the eigenvalues to be less 
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than 1. It is observed that the allowed regions by the unitarity bound for larger M is smaller than 
those for smaller M. This is because the A2 coupling increases as M becomes larger. In fact, the 
excluded regions are determined by the following unitarity condition [jjj; 

12Ai + 22A 2 + 14A 3 ± ^/(12Ai - 22A 2 - 14A 3 ) 2 + 144A|| < 16vr. (36) 

On the other hand, the vacuum stability bound becomes milder as M is taken to be a larger value 
because of the increasing A2 coupling. For a fixed value of m,H 5 and M, a larger M value is allowed 
(excluded) by the unitarity (vacuum stability) bound. 

Fig. [2] also shows the regions excluded by the unitarity and the vacuum stability conditions for 
the case of m# 3 = 300 GeV and v a = 1 MeV. The allowed regions are much smaller than those in 
the case of m# 3 = 150 GeV. The excluded regions from the vacuum stability for smaller (larger) 
values of M are determined by the third (fourth) inequality in Eq. (|35p . 

In the case of larger va values (e.g., va > 10 GeV), the regions excluded by the unitarity 
(vacuum stability) condition are larger (smaller) compared to the small va case. This is because 
the Ai coupling becomes larger. In addition, the singlet Higgs boson mass gets a larger value, so 
that the regions excluded due to uih 1 < are smaller in the larger va case. 



B. Zbb data 



The renormalized Zbb vertex is defined by 



21| 



Z^{g^P L +g R P R )b 



-Zbb 



Sy/C\v 



where the renormalized coupling g^' R can be expressed as 



9 b 



Kn L,R (SM) + fi g L,R (GM) with 



9 b +$9b 



9l 



SW^ibi 9b 



R 



s wQbi 



(37) 



where Sg^' R ^ SM ^ (Sg^' R ^ GM ^) denote the one- loop corrections to the Zbb vertices from the SM 
(GM) contributions, where the W boson and the NG boson (H 3 ) are running in the loop, If (Qf) 
is the third component of the isospin (the electric charge) for the field /, and s\y = smOyy and 



1 — s^y. The analytic formulas for 5g 



L,R (SM) 



are calculated as 



2M 



is given in Ref. 



22| , and their numerical values 



5g L (SM) 



-0.4208, 



S R (SM) 



0.0774. 



(38) 
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FIG. 3: Constraint from the i?f, data given in Eq. (fH)]) on v& as a function of m# 3 . The region above the 
black (red) line is excluded at la (2a) level. 



The one- loop correction 5g^ GM ^ is given in terms of the Passarino-Veltman function 



by 



S 9\ 



L(GM) 



e 2 tan 2 6>#m 2 1 

V 2 167T 2 



sy/cw 



1 



C2wC24(m b ,m z ,m b ,mt,mH 3 ,m H3 ) + 2s w QtC2i{m h ,m z ,m h ,m H - i ,m t ,m t ) - -s w Q 



+ m 2 t (I t - s 2 w Q t )Co(mlm%,mlm H3 ,m t ,m t ) - (I b - s^B^ml m t , m Ha ) 



(39) 



On the other hand, Sg^ GM ^ can be neglected because the corrections are proportional to the 



bottom quark mass [see Eq. 



. We can also neglect the contributions from H$ loop diagrams 



for the same reason. The renormalized couplings g b ,R can be compared to the experimental value 



of Rl xp 
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Rl xp = 0.21629 ± 0.00066. 



(40) 



In Fig. [3l we show the excluded parameter space in the rriH 3 -VA plane using the Rb data in 
Eq. (|40p . Basically, the upper bound on va increases monotonically with m# 3 . The 2a bound is 
about 25 GeV more relaxed than the la bound over the considered range. We note in passing that 



the constraint of the b — > S7 data for the GM model is similar to that in the Type-I 2HDM 
and is milder than the Rb constraint. 
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IV. HIGGS DECAYS 



In this section, we discuss the decay of the triplet-like Higgs bosons, namely the 5-plet Higgs 
bosons (= Hf or H®), 3-plet Higgs bosons H3 (= H 3 or H®) and the singlet Higgs boson 
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V A [GeV] v A [GeV] v A [GeV] 



FIG. 4: The decay branching ratios of H§ + (left), H§ (center) and H® (right) as a function of va- We take 
mH 3 = 150 GeV, rn^=125 GeV and a = in all the plots. The mass difference Am is taken to be for the 
upper three plots, and to be —50 GeV for the lower three plots. 

Hf. Decay branching ratios of the Higgs bosons depend on the mass parameters m# 5 , m# 3 and 
ruffi, the VEV of the triplet field va, and the mixing angle a. The mass of the SM-like Higgs boson 
h is fixed at 125 GeV. Using the mass relation given in Eq. (|20p . we can treat as an dependent 
parameter determined by m# 3 and m# 5 . Hereafter, we take Am = mn z — m H 5 , mH 3 and va as 
the input parameters, and assume a = for simplicity. Once we apply the mass relation, there are 
three different patterns of masses for the triplet-like Higgs bosons. In the case of Am = 0, all the 
masses of the triplet-like Higgs bosons are degenerate: m# 5 = rnjj 3 = mHi > whereas in the case of 
Am > (Am < 0), the mass spectrum is then mn x > rriH 3 > "Jh 5 ( m H 5 > m-Hs > f^ffi)- 

First, we consider the decays of the 5-plet Higgs bosons. In the case of Am > 0, the 5-plet 
Higgs bosons can decay into weak gauge boson pairs or lepton pairs depending on the magnitude 
of va- When Am < 0, the 5-plet Higgs bosons can decay into a 3-plet Higgs boson and a gauge 
boson, such as H§ + — > W + H^~ and H§ — > W + H®, in addition to the two decay modes allowed in 
the case of Am > 0. 

In Fig. 2J the decay branching ratios of H§ , H§ and are shown as a function of va in the 
case of rriH 3 = 150 GeV. When Am = (upper row), the main decay modes of , H§ and 
H% change from £ + £ + , £ + u, and vv to W+Z, and VF+VF" or ZZ at around v A = 10~ 3 
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FIG. 5: Same as Fig.gJ but for m H3 = 300 GeV. 

GeV, respectively. Here H® decays more dominantly into W + W~ than ZZ because of the mass 
threshold effect. When Am = — 50 GeV (lower row) and for the wide range of 10~ 8 < va < 1 
GeV, the main decay mode of H§ + is H^~W + , those of are Z and H®W + , and those of 
H% are H^W T and H$Z. 

Fig. [5] shows the decay branching ratios of the 5-plet Higgs bosons for mn z = 300 GeV. The 
general behavior here is roughly the same as the case with mn 3 = 150 GeV. The crossing point for 
the main decay modes in each of the upper plots (Am = 0) slightly shifts to a smaller va (~ 10 
GeV). 

Fig. [6] shows the contour plots of the decay branching ratios of H§ + , and on the va- 
\Am\ plane (with Am < 0) for the cases with m# 3 = 150 GeV (upper plots) and m# 3 = 300 GeV 
(lower plots). There are always three distinct regions in this plane. In the region of small | Am\ and 
small (large) ua, the main decay modes of the 5-plet Higgs bosons are the a pair of leptons (weak 
bosons). In the region of large |Am|, they are a 3-plet Higgs boson and a gauge boson, denoted 
by H%V in the plots (V = or Z), where it is understood that all the possible channels of H3V 
should be summed over. 

Secondly, we consider the decays of the 3-plet Higgs bosons. The 3-plet Higgs bosons can decay 
into a pair of fermions through the Yukawa interactions given in Eq. (|28[) and a pair of leptons 
through the neutrino Yukawa interaction given in Eq. (|26p , depending on the value of v a in the case 
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FIG. 6: Contour plots of the decay branching ratios of H~£ + (left column), H§ (center column) and 
(right column) on the t>A-|Am| plane (with Am < 0). We take m^=125 GeV and a = in all the plots. 
The upper (lower) three plots show the case for mjj 3 = 150 GeV (300 GeV). Each solid (dashed) curve 
represents the branching ratio of 50% (90%) for the corresponding decay mode indicated by the arrow. 



of Am ~ 0. In the region dominated by fermionic decays, the main decay mode strongly depends 
on mH-i- When rriH 3 is smaller than the top quark mass, H% {H®) mainly decays into t + v or cs 
(bb), whereas in the case of mt < mn 3 < 2mt, (H®) decays into tb (bb). Furthermore, when 
rriH 3 is larger than 2mt, decays dominantly into ti, and still mainly into tb. In addition, the 
3-plet Higgs bosons can decay into the SM-like Higgs boson h and a gauge boson, e.g., — > hW + 
and H® — > hZ if m# 3 is larger than m^. When Am > (Am < 0), the 3-plet Higgs bosons can 
decay into a gauge boson and a 5-plet (singlet) Higgs boson. 

In Fig. [3 the decay branching ratios of H% and H® are shown as a function of v& for mn 3 = 150 
GeV. The mass difference Am is taken to be 0, 50 GeV and —50 GeV in the top, middle and bottom 
plots, respectively. From the top two figures, it is seen that the dominant decay modes of (H®) 
change from l + v (vv) to t + v or cs (bb) at around v& = 10 -3 GeV. In the case of Am = 50 
GeV (middle plots) and for a wide range 10 -8 < v& ^ 10 GeV, the 3-plet Higgs bosons mainly 
decay into a 5-plet Higgs boson and a weak gauge boson, i.e., — > H^ + W~ , — > Z and 
H+ H°W + for H+ decays and H$ HfW* and H°Z for decays. On the other 

hand, in the case of Am = —50 GeV (bottom plots), the main decay modes of (H®) are H®W + 
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FIG. 7: Decay branching ratios of H$ (left column) and H® (right column) as a function of «a- We take 
to# 3 = 150 GeV, nth = 125 GeV and a = in all the plots. The mass difference Am is fixed to (top 
plots), 50 GeV (middle plots) and —50 GeV (bottom plots), respectively. 

(H$Z) in the range of 10~ 8 < v A < 10 GeV. 

Fig. [8] shows the decay branching ratios of and H® as a function of va for mjj 3 = 300 GeV. 
The mass difference Am is taken to be 0, 50 GeV and —50 GeV in the top, middle and bottom 
plots, respectively. When Am = (top plots), the main decay modes of (H®) change from £ + u 
(vv) to tb and hW + (hZ) at v A ^ 10~ 4 GeV. When Am = 50 GeV (middle plots) and Am = -50 
GeV (bottom plots), the main decay modes are the same as in the case of mn 3 = 150 GeV in the 
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FIG. 8: Same as Fig. El but for m H3 = 300 GcV. 

range of 10~ 7 < i>a < 1 GeV. 

In Fig. El we give the contour plots of the decay branching ratios of and H® for m# 3 = 150 
GeV. The mass difference Am is taken to be positive (negative) in the upper (lower) two figures. In 
this figure, BR(i7^~ — > H§V) and BR(ifg — > H$V) denote the sums of the decay branching ratios 
of the modes with a 5-plet Higgs boson and a gauge boson. BR(i/ 3 '" — > ff) and BR(i^3 — > ff) 
denote the sum of the decay branching ratios of — > t + u and H% — > cs and that of #3 — > bb 
and — > t + t~ , respectively. Similar to Fig. [6l it is seen that there are three distinct regions in 
this plane. In the small Am and small (large) v& region, the main decay modes are l^v (t + u) for 
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FIG. 9: Contour plots of the decay branching ratios of (left column) and (right column) on the 
«A-Ara plane. We take iriH 3 = 150 GeV, m^=125 GeV and a = in all the plots. The upper (lower) two 
plots show the case with Am > (Am < 0). Each solid (dashed) curve represents the branching ratio of 
50% (90%) for the corresponding decay mode indicated by the arrow. 



and vv (bb) for H®. On the other hand, in the large Am region, the decay modes associated 
with a 5-plet (singlet) Higgs boson dominate in the case of Am > (Am < 0). We notice that the 
regions where the H§V decay is dominant are wider than the corresponding one where the HfV 
decay is dominant. This is because of a larger number of decay modes in H§V . 

Fig. [10] shows the contour plots of the branching ratios of the 3-plet Higgs bosons for m# 3 = 300 
GeV. In the plots of the left column, there is no dashed curve corresponding to the branching ratio 
of 90% for the H§ — > tb decay mode. This is because the — > hW + decay mode is also 
kinematically allowed at the same time when the H% — > tb is open, and the former amounts to 
around 30%. 

We here comment on the decays of the singlet Higgs boson H®. When we take a = 0, the decay 
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FIG. 10: Same as Fig. H but for m Ha = 300 GeV. 



property of H® is similar to that of H® . In the case of Am < 0, H$ can decay into vv (JN^W or 
ZZ) for smaller (larger) values of v&. When Am > 0, Hf can decay into a 3-plet Higgs boson and 
a weak gauge boson. If a ^ 0, Hf can mix with h and can thus decay into fermion pairs via the 
mixing in addition to the above-mentioned modes. 

Throughout this section, the decay properties of the 5-plet Higgs bosons and the 3-plet Higgs 
bosons can be separately considered for four different regions in the v&-Am plane, as schematically 
shown in Fig. QTJ In Region I, all the triplet-like Higgs bosons mainly decay leptonically: 

H+ ->■ £ + v, Hi -»• vv. (41) 

In this region, the mass of the 5-plet Higgs bosons is constrained to be m# 5 > 400 GeV by the 
search at the LHC for doubly-charged Higgs bosons decaying into same-sign dileptons 11[. In 



Region II, the 5-plet Higgs bosons mainly decay into the weak gauge boson pairs, while the 3-plet 



21 




FIG. 11: Four regions are schematically shown on the i?a-|Ato| plane. 

Higgs bosons decay into the fermion pairs. When the mass of the 3-plet Higgs bosons is less than 
the top quark mass, the main decay modes are 

-> W + W + , -> W + Z, Hi -> W + W~/ZZ, 

H+ -> t + v/cs, Hi -»• 66. (42) 

For Region III and Region IV, one has to separately consider the cases whether the sign of Am is 
positive or negative. In the case of Am > 0, the 5-plet Higgs bosons mainly decay into the lepton 
pairs (weak gauge boson pairs) in Region III (Region IV). The 3-plet Higgs bosons mainly decay 
into a 5-plet Higgs boson and a weak gauge boson: 

H++ -> £+£+ (W + W + ), H+ -> £ + u {W + Z), H° -> vu (W + W~/ZZ), 

H + ^H+ + W~/H+Z/HlW + , Hl^HfWT/HlZ. (43) 

In the case of Am < 0, the main decay modes in both Region III and Region IV are 

H+ + -> H+W+, H+ -»• H+Z/HlW + , Hi -> H±W T /HlZ 

i? 3 + -»• i? 1 °VF + , ^3 -> fljz. (44) 

V. PHENOMENOLOGY AT THE LHC 

In this section, we discuss how the custodial symmetry of the GM model can be tested at the 
LHC. There are characteristic features of the triplet-like Higgs bosons that mainly originate from 
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the triplet field A of the model. (1) The masses of the Higgs bosons belonging to the same SU(2)y 
multiplet are the same. (2) The 5-plet and the singlet Higgs bosons have the Gauge-Gauge-Scalar 
type of couplings as listed in Table IIIIl but not the Yukawa couplings given in Eq. (j28|) , while the 
3-plet Higgs bosons have the Yukawa couplings, but not the Gauge-Gauge-Scalar type of couplings. 
These features can be used to test the custodial symmetry of the GM model. 

As is discussed in the previous section, it is important to study the decay pattern of the triplet- 
like Higgs bosons. In particular, feature (2) mentioned above can be most clearly tested in Region II 
because the 5-plet (3-plet) Higgs bosons mainly decay into weak gauge boson pairs (fermion pairs) 
in this region. In the following discussion, we focus on Region II and the detectability of the 5-plet 
and 3-plet Higgs bosons. 

A. Production modes 

There are several production modes for the 5-plet Higgs bosons H§ and the 3-plet Higgs bosons 
H3, as listed below. Throughout this section, q,q',Q,Q' and those with bars denote light quarks 
and anti-quarks. 

1. The Drell-Yan process 

if 5 and H3 can be produced in pairs via 7 and Z, e.g., pp — > and pp — >• H3H3. The 

cross section is determined by the gauge coupling as well as the Higgs masses rriH 5 and rriH 3 , 
independent of the value of v&. 

2. The mixed Drell-Yan (mDY) process 

H§ and H3 can be produced at the same time, e.g., pp — > H5H3, which we call the mixed 
Drell-Yan (mDY) process to be separated from the usual Drell-Yan process mentioned above. 
The cross section is proportional to c 2 H , and is thus relatively suppressed in comparison with 
the Drell-Yan process, especially in the large va case. 

3. The weak vector boson fusion (VBF) process 

The single production of occurs via the qQ — > H§ process. The cross section is propor- 
tional to v\, so that this mode can be important in the large va case. 

4. The weak vector boson associated process 

In addition to the VBF process, H§ can also be produced in association with a weak gauge 
boson, e.g., qq' — > H§V. The cross sections of such modes are proportional to v\ as for the 
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FIG. 12: Production cross sections of the 5-plet Higgs bosons in various processes as a function of mjy 5 . 
The upper-left, upper-right, and bottom plots show the production cross sections for H^ 1 , and H®, 

respectively. In all the plots, we take ua = 20 GeV and Am = 0. The LHC collision energy is assumed to 
be 8 TeV (dashed curves) and 14 TeV (solid curves). 

VBF production mode. Thus, this mode can also become important when the VBF process 
is important. 

5. The Yukawa process 

H3 can be produced through the Yukawa interactions given in Eq. f|28[) as the gluon fusion 
process for the SM Higgs boson: gg — > H®. There are t-channel H% and H® production 
modes: gb — > tH^ and gb — > bH®. These production cross sections are proportional to 
tan 2 6h- 
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6. The top quark decay 

When rriH :i is smaller than the top quark mass, H% can be produced from the top quark 
decay. The decay rate of the t —> bHf depends on tan 2 6h- 

Among these production processes, channels 3 and 4 can be useful to discriminate the GM 
model from the others with doubly-charged Higgs bosons and to test the mass degeneracy of H§. 
In the HTM, for example, the doubly-charged Higgs boson can in principle be produced via the 
VBF and the vector boson associated processes. However, these cross sections are much suppressed 
due to the tiny triplet VEV required by the electroweak rho parameter. In the GM model, va can 
be of order 10 GeV, so that these production processes become useful. The mDY process is also 
a unique feature of the GM model because the Higgs bosons H§ and H3 having different decay 
properties are produced at the same time. In particular, when Region II is realized, the main decay 
modes of these two Higgs bosons are distinctly different. Thus, this process can be useful not only 
to test the mass degeneracy of H3 but also to distinguish the model from the others also having 
doubly-charged and/or singly-charged Higgs bosons. 

In Fig. [T2l production cross sections of the 5-plet Higgs bosons from channels 2, 3, and 4 are 
shown as a function of mu 5 for the LHC running at 8 and 14 TeV. We take va = 20 GeV and 
Am = as an example in all the plots. It is noted that the dominant production mechanism is 
the VBF process for a sufficiently large m# 5 . 



B. Signal and background analysis 

We will first discuss the VBF process and the vector boson associated process to study the mass 
degeneracy among H^, Hf and flg. Then we turn to the mDY process. 

Let us consider the case with m# 3 = 150 GeV, Am = 10 GeV (i.e., m# 5 = 140 GeV) and 
va = 20 GeV as an example in Region II. In this case, the 5-plet Higgs bosons decay into gauge 
boson pairs almost 100% (the branching fractions of H® — > W + W~ and H® — > ZZ being 67% and 
33%, respectively). On the other hand, Hf decays to r^v at 66% and cs at 29%, and H® decays 
to bb at 89%. We note that the branching fraction of t — > H^b here is around 0.4%. The upper 
limit of the top quark decay into a charged Higgs boson and the bottom quark is 2-3% in the case 
where the charged Higgs boson mass is between 80 and 160 GeV, under the assumption that the 



charged Higgs boson decays to rv at 100% 



Thus, the selected parameter set is allowed by 



the constraint from the top quark decays. The signal events from the VBF production processes 
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for the 5-plet Higgs bosons are given by 



gQ -> i^g'Q' -> 

gQ -> flggV -> W^W^jj/ZZjj. (45) 
From the vector boson associated processes, we have the following events 
qq' -> -> W^W^ij 

gg -> -> W ± Zjj, qq' -> -> W ± Zjj, 

^ ^ F 5 °Z -> W+^-ij/ZZji, gg' ->• flgw* -> W+W-jj/ZZjj, (46) 

where the associated weak gauge bosons are assumed to decay hadronically so that they have 
the same final states as the VBF process. Moreover, we consider the case where the weak gauge 
bosons produced from the decay of H$ decay leptonically. Then the final states of the signal events 
have same-sign (SS) dileptons plus dijets and missing transverse energy (i^l^jj^j-p) for the 
production mode, where I denotes collectively the light leptons e^- and ^ hereafter. The final 
state of the H 5 production mode includes trileptons plus dijets and missing transverse energy 
{£ £ £^jj$j<), while that for the H® production mode has opposite-sign (OS) dileptons plus dijets 
and missing transverse energy (jP 1 £ J ^jj^) T ). The corresponding background events for these signal 
events are from the W^W^jj for the production, W^Zjj for the H§ production, and tt, 

W^W^jj and ZZjj for the H® production. 



We simulate the signal and the background event rates by using MadGraph 5 [29|] at the parton 
level for the cases where the LHC operates at the center-of-mass (CM) energy y/s of 8 TeV and 14 
TeV. We impose the following basic kinematic cuts 

jP T > 20 GeV, p e T > 10 GeV, \rf\ < 5, \rf\ < 2.5, AR jj > 0.4, (47) 

where p 3 T and p l T are the transverse momenta of the jet and the lepton, respectively, rf and rf 
are the pseudorapidities of the jet and the lepton, respectively, and AR n is the distance between 
the two jets. The cross sections for the signal and background events are listed in Table [J where 
the signal cross section includes contributions from the VBF production and the vector boson 
associated production. An integrated luminosity of 100 fb _1 is assumed in the simulations. In this 
table, the signal significance is defined by 

S = S/VS~+B, (48) 
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(8.17) 
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6-jet veto 














0.48 
(1.36) 


1.82 
(10.8) 


3.16 

(3.90) 



TABLE I: Signal and background cross sections in units of fb after each kinematic cut, along with the 
significance S defined by Eq. (|48|) based on an integrated luminosity of 100 fb -1 . The numbers without 
(with) parentheses correspond to the case with a CM energy of 8 TeV (14 TeV). The signal cross section 
includes contributions from both the VBF production and the vactor boson associated production processes. 
For the t^Pjjflj, events, we further impose the requirement of the &-jet veto for each jet to reduce the 
background, where the ^-tagging efficiency is take to be 0.6 31 1. 



where S and B are the numbers of the signal and background events, respectively. The significance 
of the event from the Hj^ production process exceeds 5 even using simply the basic 

cuts. However, the significances for the remaining two events from the H§ and H® production 
processes are less than 1. For the jj$T event, in particular, the background is larger than 
the signal by 3 to 4 orders of the magnitude because of the huge tt background. 

To improve the significance, we need to impose additional kinematic cuts. Fig. [13] shows the 



30] in 



distributions of the pseudorapidity gap Arfi for the dijet system and the transverse mass 
the leptons plus missing transverse energy system for y/s = 8 TeV and the integrated luminosity 
of 100 fb -1 . Explicitly, these two kinematical quantities are defined by 

2 



- 2 









rJl 



rJ2\ 



(49) 
(50) 



where M^- 1S and are the invariant mass and the vector sum of the transverse momenta of 
the charged leptons, respectively, and p is the missing transverse momentum determined by the 
negative sum of visible momenta in the transverse direction. In Fig. [1~3"1 the distributions of Ar/^ 
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FIG. 13: Arfi and Mt distributions for the signal and background events. The top, middle and bottom 
plots show these distributions for the £ ± £ ± jj$Ti ^ ± ^ ± ^ T 33 tr an d t^^jjtr events, respectively. The 
distributions for the signal events are divided into those from the VBF process and vector boson associated 
process (green dashed curve), the mDY process (blue dashed curve) and the sum of them (red solid curve). 
The bin size for the Arjjj (Mt) distribution is taken to be 0.2 (5 GeV). The integrated luminosity and the 
CM energy are assumed to be 100 fb _1 and 8 TeV, respectively. 

and Mt for the signal events from the VBF process plus vector boson associated process and the 
mDY process are separately indicated by dotted lines. The latter production mode will be discussed 
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in details later. A significant feature of the VBF process is that the two external quark jets are 
almost along the beam direction and carry most of the energy of the collider protons. Therefore, 
they are mostly detected in the forward regions. This is seen in the Arf^ distribution of Fig. [131 
The end point in the Mt distribution of signals rests at around 140 GeV, corresponding to the 
mass of the 5-plet Higgs boson. 

According to the above-mentioned observations, we find the following additional kinematic cuts 
useful in further reducing the backgrounds: 

At7">3.5 (>4.0 for ^ T jj$ T ), 50 < Mt < 150 GeV. (51) 

The cross sections for the signals and backgrounds in each step of the kinematic cuts are listed 
in Table HI After making the first two cuts, the significances of the events from and H§ 

achieve 13.2 and 5.2, respectively. However, the significance of the events from H® is around 1.4. 
We further require that events with at least one fr-jet are tagged and rejected in order to reduce 
the ti background. The 6-tagging efficiency is taken to be 0.6 By using this cut, the tt 

background events with the final state of bb£ + £~ $t can be reduced to be 16%. Consequently, the 
signal significance for the £ + £~jj$T event can reach 3.16 (3.90) with y/s = 8 TeV (14 TeV) after 
all the cuts discussed above are imposed. 

Next, we focus on the mDY production mode discussed in the previous subsection. In order to 
reconstruct the masses of H$ Higgs bosons, we consider their hadronic decays, namely H$ —> cs 
and H® — >• bb. The signal events are 

pp -> H^H^ -> W ± W ± cs, 

pp -> HfH^ -»• W ± Zcs, pp -»• HfHl -> W ± Zbb, 

pp -> H^Hf -> W + W~cs/ZZcs, pp -> H®H® -> W + W-bb/ZZbb, (52) 

where leptonic decays of the weak gauge bosons from the H§ decays are also assumed in this 
analysis. Thus, the final states of the signal events from the mDY process are the same as those 
from the VBF process as well as the associated process. Its difference from the VBF process is 
observed in the Arf 3 distribution of the dijet system. In the mDY process, the dijets in the final 
state come from the decay of the 3-plet Higgs boson, not the external quark jets. According to the 
plots in the left column of Fig. [T3l the events from the mDY process concentrates in the Aif 3 < 2.5 
region for all the three cases. On the other hand, the Mt distributions from the mDY process and 
the VBF plus associated process are almost the same. This is because the leptons plus missing 
transverse energy system come from the decays of H§ in both processes. Therefore, we apply the 
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Cuts 






W ± W ±jj 


S 






W ± Zjj 


S 


Basic 


3.71 (8.72) 


0.72 (1.63) 


3.48 (8.13) 


15.8 (24.1) 


0.61 (1.60) 


0.53 (1.21) 


45.9 (1.39xl0 2 ) 


1.66 (2.36) 


M T 


3.65 (8.57) 


0.71 (1.60) 


1.02 (2.20) 


18.8 (28.9) 


0.61 (1.60) 


0.53 (1.21) 


1.16 (3.42) 


7.52 (11.3) 



TABLE II: Signal and background cross sections in units of fb after each kinematic cut, along with the signif- 
icance based on an integrated luminosity of 100 fb . The numbers without (with) parentheses correspond 
to the case with a CM energy of 8 TcV (14 TeV). 



same Mt cut given in Eq. (|5ip to this analysis, but not the Aif 3 cut. In the analysis of the 
mDY process, the l^P jj$T signal events are overwhelmed by the huge background from the ti 
production. 

Table [TT] lists the cross sections of the signal and the background events after imposing the basic 
cut and Mt cut. In addition, the signal significance is given by assuming an integrated luminosity 
of 100 fb -1 . We find that the significances exceed 5 in both cases after imposing the My cut. 

Fig. HH show the dijet invariant mass Mjj distributions of the signal and the background events. 
These distributions are plotted after imposing the Mt cut. We can see a peak at around 150 GeV, 
corresponding to the mass of the 3-plet Higgs bosons, in both the i^i^jj^T an d jj$T 
events. This suggests that the mass degeneracy between H% and H® can be readily established 
from the mDY process. First, the i^i^jj^T event comes from the H^H^ production. Thus, the 
peak at around 150 GeV in the Mjj distribution gives the mass of Hf. Secondly, the l ± £ ± t^jj$ T 
event comes from the H^H^ and H^H® production processes. These two production cross sections 
are almost the same as shown in Fig. [12j Nevertheless, the decay branching fractions of — > cs 
and H® — > bb are about 30% and 90%, respectively. Thus, the i^i^i^jj^T event mainly comes 
from the H^H® production. Therefore, one can conclude that the peak at around 150 GeV in the 
Mjj distribution is the mass of H®. 

VI. HIGGS TO 77 AND Z 1 DECAYS 

In this section, we discuss the decays of the SM-like Higgs boson h to diphotons and the photon 
plus Z boson, both of which are loop-mediated processes in the SM. The Higgs to diphoton decay 
is one of the most important modes in the Higgs boson search at the LHC. According to the current 
data, the signal strength, defined by (the observed cross section) /(the expected cross section in 
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FIG. 14: Invariant mass distribution for the dijets system. The bin size in this distribution is 5 GeV. The 
distributions for the signal events are divided into those from the VBF process and vector boson associated 
process (green dashed curve), the mDY process (blue dashed curve) and the sum of them (red solid curve). 
The integrated luminosity and the CM energy are assumed to be 100 fb _1 and 8 TeV, respectively. 

the SM), of the diphoton mode is 1.6 ± 0.4 at the CMS fl and 1.8 ± 0.5 at the ATLAS Q. It is 
consistent with the SM prediction at the 2a level. If the observed deviation persists, it may hint 
at contributions from new charged particles that can couple to the SM Higgs boson. The Higgs 
decay into the photon and Z boson is also important to determining the structure of the Higgs 



sector 



34j | . The decay rate of this mode is closely related to that of the Higgs to diphoton mode in 



the sense that particles contributing to the latter generally also contribute to the former. Yet the 
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34| . In the GM model, the doubly-charged 



deviations do not follow the same pattern in general 
Higgs boson as well as the singly-charged Higgs bosons H 5 and can contribute to these 
processes in addition to the W boson and the top quark at the one-loop level. The decay rates of 
these processes are calculated as 



T(h -> 77) 



I V- g| W / (m fl ) + - E Q 2 f N ih/2 + (c a c H + 2 -^s a s H )h 



128^/2 



7T } 



s 



CH 



f 



(53) 



T(h -> Z-y) 



\f2G F a 2 cm m? h 



128vr 3 



■m. 



2^6 



y~] Qs^hss*gzss* ch H —S a SH)Jl 



CH 



(54) 



where iV"/ = 3 (1) for / = q (£), and the loop functions /q, 1/2,1 f° r h — > 77 and Jo,i/2,i Ior h — > Zj 
are given in Appendix C. The summation over S includes H+ + , H+ , and Ht . In Eqs. (EHD and 
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o4|) . the couplings between h and the charged Higgs bosons XhSS* are given by 
Kh++h-- =^| C ^( 3 ^3- 2M 1) + Vf7f [2m% 5 +ml-Mi + 3c? H (M?-2m% 3 )]^, (55) 

X hH+H? = ~ X hH++HZ- ( 56 ) 

(57) 



*5 11 5 

1 

7 .2 



^ (2<&i7& a + s 2 H m 2 h ) + ^ffl ( 2 4m^ + c 2 H m 2 h - M 2 ) 



and those between the Z boson and the charged Higgs bosons gzss* are given by 

9ZH++H~- = ~ 2s w)i 9ZH+Hs = g ZH+Hz = ~2 9 ZH++H^-- ( 58 ) 

To illustrate how the event rates of h — > 77 and h — > Z7 deviate from the SM predictions, we 
define the following ratios: 

R = o{gg h) G M x BR(h -)■ 77) G m r = crjgg h) GM x gjR(/i -» Z^) GM 
77 £1(55 -»• /i) SM x BR(h -> 77)sm ' Z7 <r(ff5 -> /i)sm x Bfl(/i -»• ^7)sm ' 

where a(gg — > /i)sM [c(<?<? — > ^)gm] is the gluon fusion production cross section in the SM (GM 
model), and BR(h — > X)sm [BR(h — > X)qm] is the branching fraction of the h — > X decay mode 
in the SM (GM model) with X = 77 or Z7. In fact, the ratio in the production cross sections, 
v(99 -> h) GM /a(gg -> h) SM , can be replaced by c 2 Jc 2 H . 

In the numerical calculation of i? 77 and i?^ 7 , we use the parameterization given in Eq. ([33]). 
The mass of the singlet Higgs boson mu\ does not directly affect the decay rates of h — > 77 and 
h — > Z-f. Nevertheless, it affects the parameter space as constrained by the vacuum stability and 
unitarity conditions. In this parameterization, the couplings given in Eq. (|57p can be rewritten as 

2 1 

X hH++HZ- = Z2( m h ~ M2 )> X hH+H~ = ~—^2 ( 2c H m H :i + S H m h)- (60) 

Fig. [15] shows the contours of i? 77 (left plot) and Rz^ (right plot) on the M-rriH 5 plane. Here 
we take m Hz = 150 GeV, M = 300 GeV and a = 0. The triplet VEV v& is taken to be 20 GeV in 
the upper two plots and 60 GeV in the lower two plots. The blue, pink and gray shaded regions are 
excluded by the vacuum stability bound, unitarity bound and by having a negative value for mn x , 
respectively. For fixed values of mji 5 , both i? 77 and Rz-y increase with M. In other words, there 
is a correlation between the two ratios in this model. For the case with a larger v&, i? 77 and i?z 7 
tend to have smaller values because the hW + W~ coupling gets smaller. Using mjj 5 = 150 GeV as 
an example, the maximally allowed values of i? 77 and Rz-y are about 1.8 (1.0) and 1.2 (0.8) in the 
case of va = 20 GeV (60 GeV), respectively. 
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FIG. 15: Contour plots of R 11 (left) and Rz 1 (right) on the M-ttih 5 plane in the case of m# 3 = 150 GeV, 
M = 300 GeV and a = 0. In the upper (lower) two plots, v A is taken to be 20 GeV (60 GeV). The blue, 
pink and gray shaded regions are respectively excluded by the vacuum stability bound, unitarity bound and 
by having a negative mass for Hi . 



Fig. [16] also shows the corresponding contour plots for the case of m# 3 = 300 GeV, M = 200 
GeV and a = 0. The parameter space allowed by the unitarity and the vacuum stability constraints 
is much smaller than the previous case. Again, using mn 5 = 150 GeV, the maximally allowed values 
of R-y-y and Rz-y are almost the same as the case of m# 3 = 150 GeV, but the minimum values of 
both i? 77 and Rz^ are around 1.0 in the case of v& = 20 GeV. 
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FIG. 16: Contour plots of R 11 (left) and Rz 7 (right) on the M-ttih 5 plane in the case of m# 3 = 300 GeV, 
M = 200 GeV and a = 0. In the upper (lower) two plots, v A is taken to be 20 GeV (60 GeV). The blue 
and pink shaded regions are respectively excluded by the vacuum stability bound and unitarity bound. 

VII. CONCLUSIONS 

We have discussed how to test the custodial symmetry in the Higgs sector of the GM model at 
the LHC. This can be done by experimentally verifying three characteristic features. First, there 
are several Higgs bosons in addition to the SM-like Higgs boson h; namely, a pair of doubly-charged 
Higgs bosons, two pairs of singly-charged Higgs bosons, a CP-odd Higgs boson and three CP-even 
Higgs bosons. These Higgs bosons can be classified into the 5-plet Higgs bosons , Hf , H®) , 

the 3-plet Higgs bosons (H^,H®), and the singlet Higgs boson H® under the custodial SU(2)y 
symmetry. The Higgs bosons belonging to the same SU{2)y multiplet have the same mass, subject 
to small electromagnetic corrections at the order of a few hundred MeV. Secondly, the 5-plet and 
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the singlet Higgs bosons can couple to weak gauge boson pairs, but not fermion pairs via the usual 
(not neutrino) Yukawa interaction at the tree level. On the other hand, the 3-plet Higgs bosons 
can couple to fermion pairs, but not weak gauge boson pairs. As discussed in the main text, such 
a feature leads to specific final states for detecting these Higgs bosons and measuring their masses. 
Thirdly, the VEV of the isospin triplet Higgs fields can be taken to be of order 10 GeV while 
keeping p = 1 at the tree level. This is not possible in models with triplet fields in general. 

The decay properties of the triplet-like Higgs bosons have been discussed in details. They 
depend on the mass splitting Am, defined by rriH a — m/( 5 , and the triplet VEV vj\. We find that 
the parameter space in the v^-Am plane can be divided into four regions, among which the main 
decay modes of the triplet-like Higgs bosons are quite distinct. 

We have discussed the collider phenomenology of the GM model at the LHC in Region II where 
the 5-plet Higgs bosons mainly decay to weak gauge boson pairs, whereas the main decay modes 
of H% are tv and cs and that of H® is bb when the mass of the 3-plet Higgs bosons is less than mj. 
We focus on the VBF, the vector boson associated and the mDY production processes in order to 
verify the custodial symmetric nature of the model. We find that H 5 and can be detected at 
more than 5a level by using the forward jet tagging for the VBF process and the transverse mass 
cut on the charged leptons and missing transverse energy system if the center-of-mass energy and 
the luminosity are 8 TeV and 100 fb _1 , respectively. The significance of the H® Higgs boson can 
be reached at 3a level by further imposing the 6-jet veto. 

We also find that the 3-plet Higgs bosons can be detected via the mDY production process. 
After the cut, the masses of H% and H® can be measured from the peak in the invariant mass 
distribution of the dijet system. Therefore, the respective mass degeneracy in the 5-plet Higgs 
bosons and the 3-plet Higgs bosons can be tested. 

We have also investigated the h — > 77 and h — > Zj processes in the GM model. In this model, 
the and bosons can contribute to these processes in addition to the SM top quark 

and the W boson at one-loop level. We find that in the parameter space consistent with the 
unitarity and the vacuum stability, the maximally allowed value of i? 77 is around 1.8 (1.0) for the 
parameter choice of m# 3 = 150 GeV, m# 5 = 150 GeV and v& = 20 GeV (60 GeV). Deviations 
in the rates of h — > Z7 and h — > 77 processes from the SM predictions can be used to distinguish 
models with various extended Higgs sectors. In the GM model, the maximally allowed value of 
R Zl is around 1.2 (0.8) for m H:i = 150 GeV, m Hs = 150 GeV and v A = 20 GeV (60 GeV). For the 
cases of larger m# 3 {e.g., rrtn z = 300 GeV), the maximally allowed values of i? 77 and Rz^ are not 
so different from the case of m# 3 = 150 GeV. But, the minimum values of both i? 77 and Rz-y are 



35 



about 1.0 when m# 5 = 150 GeV and t>A=20 GeV. 
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Appendix A: Relationships among different representations of the Higgs fields 

The Higgs fields expressed in Eqs. (pQ) and (12ip are related as follows: 

tr($ t $) = 2$<j>, (Al) 
tr(AtA) = 2tr( X t X + e t e), (A2) 
tr[(AtA) 2 ] = 6[tr(x f x)] 2 - 4tr[( X f x) 2 ] + 2tr(£ 4 ) + 4tr( X £)tr(£x t ), (A3) 

tr U t tr ( At *" A * 6 ) = ^U ] x)m + h-c] + 2(0+x)(xV) " (^V)tr(x f x), (A4) 

tr U\ T -1 $0 (pt A P) a6 = -±=fd<t> + \{$x4> + h-c), (A5) 

tr(Att a At 6 )(ptAP) ab = 6V2tr( X t xO- (A6) 

Appendix B: Coupling constants between the triplet-like Higgs bosons and the weak gauge 

bosons 



The Gauge-Gauge-Scalar vertices and the corresponding coefficients are listed in Table [TTTl The 
Gauge-Scalar-Scalar vertices are listed in Table \IV\ where p\ and p2 are respectively the four- 
momenta of the first and second particles in the vertex column into the vertex. 



36 



Vertex 


Coefiicient 


Vertex 


Coefficient 






H\Z^Z V 


2 

— f§ (3s a CH - 2\f§c a s H )vg slv 






hw+w- 


^-(3c a c H + 2\fQs a s H )vg tlv 


HlW+W- 


2 


kZ^Zy 


2 

■jf (3c a c H + 2y/6s a s H )vg fJ , l , 




-m SHVg ^ 




±em w g fiu 


H°W+W- 


-^-(3s Q c ff - 2\ r &c a s H )vg^ 


G^W^Zv 


Teswrnzg^u 



TABLE III: Gauge-Gauge-Scalar vertices and the associated coefficients. 



Vertex 


Coefficient 


Vertex 


Coefficient 




2e(pi -_p 2 ) M 








-e(pi -P2)p. 




^g(pi -P2) ll 


H t H 3 A H 


-e(pi ~P2)n 






G+G-A„ 


-e(pi -pz)^ 




Ti§c H (pi -P2)n 




-3-(l-2s 2 w )(p 1 -p 2 ) lx 




-^9 c h(pi ~P2)n 


H+H-Z, 


-^f(1-2^)(Pi-»^ 




T«f (pi -P2)ft 








£{2y/6c H c a + 3s H s a )(pi - p 2 ) fl 


H 5 H 3 Z V 


±^c ff (pi -p 2 ) M 


HfhWJ 


^(2\fQc H s a - 3s H c a )(pi ~P2)n 






G ± hW^ 


|(3c Q c ff + 2v / 6s ct s_ff) 


HlHlZ^ 


-^(21/60//^ + 3s H s Q )(pi - p 2 ) M 




-i^-{2y/6c H s a - 'Ss H c a )(p 1 -p 2 )n 



TABLE IV: Gauge-Scalar-Scalar vertices and the associated coefficients. 
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Appendix C: Loop functions in the h — > 77 and Z7 decay rates 

The loop functions appearing in the calculations of t he SM-like Higgs boson decay to diphotons 
are given in terms of the Passarino-Veltman functions 



25|] as 



2v 



Io(m) = — 2"[1 + 2m C (0, 0, m h , m, m, m)], 



fflf 



! l/2 



-Ami 



4m, 



h = 2m 



mr 



6 



mi 



C (0,0,m 2 h ,m f ,m f ,m f ) 



,2 i" ^,2 



+ 6 



2m 



w 



m. 



1 C (0, 0, m\, m w ,rn w , m w ) 



(CI) 
(C2) 
(C3) 



Those for the h — > Z7 process are given by 
2 



J (rrc) 



m 



J 



1/2 



e(m^ — m|) 

svyCH/(m^ - m|) L 



1 + 2m a C (0, ml, ml, m, m, m) + 9 (B (mi,m, m) - B (mi, m, m)) 

m h- m z 



2 + (Am 2 f - ml + m 2 z )C (0,m 2 z ,ml,mf,mf,mf) 



+ 



{ B o(™>h> m f> m f) ~ B o{m 2 z ,mf,m f )) 



m\ - m| 



(C4) 



(C5) 



Ji 



2m^ 



SffC^fm^ - m|) 
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2m^ 



Mr 



1 + 



2m^ 



1 + 2m^/Co(0, m 2 z ,m 2 h , mw,mw, mw) + 



m~. 



m 2 h -m 2 w 



(B (m 2 h ,m w ,m w )-B (m 2 z ,m w ,m w )) 
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(C6) 
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